A fundamental challenge in wireless communication is the unreliable and hostile nature of wireless channels. A problematic wireless channel distorts and corrupts a transmitted signal to such a degree that it is may be virtually unrecognizable at a receiver. This is especially true when a line of sight (LOS) condition is not achieved and the signal reaches the receiver after multiple reflections and scatterings. The dispersive nature of the wireless channel also creates inter-symbol interference (ISI), which cannot be mitigated by changing RF system parameters such as frequency plan, transmit power and antenna pattern.
ISI occurs in an 8-PSK (8-state Phase Shift Keying) mode of EDGE (Enhanced Data for GSM Evolution) not only due to the dispersive nature of the channel, but also due to a transmit filter. An EDGE transmit filter restricts a signal's bandwidth to approximately 200 kHz to ensure compatibility with a spectral mask previously defined for GMSK (Gaussian Minimum Shift Keying) transmission. An ISI-free Nyquist filter in the case of EDGE would lead to a signal with a 271 kHz bandwidth commensurate with the symbol rate of EDGE/GSM (Global System for Mobile communication). However, since the 8-PSK transmission is restrained to the 200 kHz bandwidth, Nyquist's ISI free sampling criteria is violated. This results in a transmission with ISI even in a non-dispersive channel when LOS is achieved.
The ISI contained within the corrupted signal can be partially mitigated by passing the signal through an equalizer such as a Linear Equalizer or a Decision Feedback Equalizer (DFE). However, neither the linear equalizer nor the DFE are optimal in terms of minimizing the symbol error rate.
A non-linear equalizer such as a Maximum Likelihood Sequence Estimator (MLSE) based on the Viterbi Algorithm (VA) is considered to be optimal, and results in the smallest error rate for a linear dispersive channel, such as the wireless channel, or a channel associated with a twisted-pair copper wire. A drawback of the MLSE is the complexity of the VA, which increases exponentially with the equalization depth and the constellation size of the modulation. In the case of a typical urban environment, the dispersive nature of the channel requires an equalization over 3 to 4 symbols which corresponds to a delay spread of 11 to 14 seconds. The number of states associated with the VA in the MLSE is 8 to 16 states for GMSK modulation and 512 to 4096 states for 8-PSK modulation. The sheer large number of states required in the case of 8-PSK modulation makes the use of MLSE computationally difficult or prohibitive.
Alternative equalizer structures such as Delayed Decision Feedback Sequence Estimator (DDFSE) and Reduced State Sequence Estimator (RSSE) have been shown to have near optimal performance with a manageable level of complexity. In the case of DDFSE, a tradeoff between complexity and performance can be achieved by truncating the depth of equalization with decision feedback ISI cancellation. RSSE allows for an even finer tradeoff between the complexity and performance by using set partitioning in which one or more modulation symbols are mapped onto a partition. It has been shown that both DDFSE and RSSE can achieve a reasonably close performance (e.g. 0.5 to 1.0 dB) in terms of Bit Error Rate (BER) when compared to the optimum MLSE equalizer.
To improve the performance of the forward error correction (FEC) at the receiver, soft likelihood values are provided to the decoder instead of hard bit decisions. The bit-wise soft log-likelihood value is defined as:
                    L        =                              log            10                    ⁡                      [                                          Pr                ⁢                                  {                                      b                    =                                          +                      1                                                        }                                                                                                              ⁢                                  Pr                  ⁢                                      {                                          b                      =                                              -                        1                                                              }                                                                        ]                                              (        1        )            where Pr{b=+1} is the probability of a bit being equal to +1, and Pr{b=−1} is the probability of a bit being equal to −1. Although the generation of likelihood values requires extra computational complexity, the performance benefits in terms of frame error rate (FER) or block error rate (BLER) are considerable. Unlike hard decisions, which are derived from a trellis with a minimum metric, soft likelihood estimation requires a comparison of the trellis with minimum metric to other trellises with larger metrics. Optimum soft likelihood information is produced by symbol-by-symbol, maximum a-posteriori (MAP) decoding algorithms. However, less computationally intensive algorithms such as Soft Output Viterbi Algorithm (SOVA) are used more commonly for MLSE equalizers.
A challenge in the implementation of RSSE-based equalizers is the generation of the soft likelihood values. In a RSSE, some of the non-surviving trellises required for the calculation of the soft likelihood value are missing. This poses a hurdle for implementing a RRSE equalizer for EDGE systems with an acceptable degree of performance. In a recent study, an RSSE-based equalizer for EDGE was proposed with soft symbol and soft bit estimation based on partial re-growing of the trellises. This algorithm looks at the existence of the non-detected path metric for a given bit value, if the non-detected path is missing. The missing branch metric and thus the corresponding path metric are calculated from a state history. The re-growing process adds computational complexity but the overall complexity of the algorithm is far less than for MLSE.
Existing RSSEs have a soft decision performance that is relatively poorer than their hard decision performance. This effect becomes more prominent as the modulation alphabet size increases or as the number of states in the equalizer decreases.